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ABSTRACT 



This paper demonstrates that the axis between cognition and 
multimedia language teaching is very short by introducing three fundamental 
concepts: (1) mediatizing, i.e., the direct use of any medium as a way of 

expressing oneself; (2) multimediatizing, i.e., the choice of a multimedia 
support to express a didactized informational content; and (3) 
re -multimediatizing, i.e., follow-up of learners' activity and appropriate 
modification of courseware. The LAVAC (Laboratoire Audio-Visuel 
Actif-Comparatif [Audio-Visual Active-Comparative Laboratory] ) system, 
courseware developed in France, which includes a student interface and 
teacher software for distance and real-time tutoring in a computerized 
laboratory, is described. (Contains 11 references.) (MES) 
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Abstract: Cogmton and multimedia language teaching seem to be at the two extremes of a 
very long axis. But the evolution of multimedia today makes it necessary to clearly define 
some seemmgly transparent cognitive notions and fimctions that may otherwise deceive the 
teacher m his exp^tions. Furthermore, teachers are the only entitled persons to Hpcip n 
Murseware ^h^on after a very precise analysis of their own didactic needs. The question 
to be asked then is "how?", and the answer to that question may not be easy. 

We shaU try to demonstrate in this p^ that this axis is in feet very short by introducing three 
fimdmental concepts. The concept of Multimediatizing is one of them and^les 
sanstactory courseware design by non-programming teachers. 



Introduction 

nWp trt t^her!", you would say to ^logize for your rather short studies that make you 

cliLw^^ speak and as a whole master a foreign language so that you can be entitled to teach it Basic^y 
1!^ 1 ^ even if this presenter role should not be excluded at a particular poiirtof 
teachmg ^edule, e.g. at the end of a leammg cycle for knowledge assessment or reinforcement puiroses 
know it teac^g and particul^ly second-language teaching is more than that and most ^hers 

amoimt of i^nal skiUs and knowledge competence necessary to achieve this sosialled 
snnple goal may somewhat be impressive. 

but ^ f ^ ^ particular language 

rae*^ underlymg mechanisms of language activity at a level common to both theSL 

aware ** la^page teacher for science students for instance must then be sufficiently scientifically 

a^of the Mo^^ content earned by the language to be able to communicate with limers in theh 
s^cc^eciahz^ fidds. After a time, an L2 teacher in a science university in France must have an operational 
co^etence in sciCTtfic domams such as physics, astronomy, mafiiematics, electronics, computer science and 
technolo^ ^ce 1^ OT her stud^ need to master these subjects in English for their future j^. 

or teaching" refers to linguistics and to the different specific purposes 

or^^ty doi™ “VO ved (htoature and dvilization are two of them), the second element is "tead^TFar 
setpnop if <aste, common sense and conviction should be put through it, teaching is a 

^ ^ observatioa The object to be studied is the^Lg Stion. 

Learners and the leammg envmmment make up this situation- ^ 

^Jproach should thus focus on the learner’s needs. The individual needs will be 
iMed and ^yzed ft^ the triple point of view of language knowledge, language use STsTiS 
p^^s defined m each learner’s syllabus (understanding science texts wiU not the same ^ of 

. .peoifie «3cbi.g 

Moreover the learning environment will set the place and time for learning 

« ridiculously sh^ if you consider the 20 to 
a umversity student has in France. Didactic theories show that W to do things is then more 

■ f^Suage acquisition cannot be intuitive in that instance as this could easUy be 
possible with 1 0 hours a day m the country of the language. ^ 
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Place was traditionally the classroom. In a class of 30 to 70 students, the syllabus has to suit a 
hypothetical virtual student that would be the average of all of them and therefore would te appropnate to ncme 
of thfftn Moreover, the teacher is the only language provider. Oral language production on behalf of the 
students is then reduced to a few seconds per hour, when it takes place. . . 

Technology changed these place and time constraints. Distance-learning is possible through the 
Internet and even in the language lab in the presence of the tether, computerized learning induced a new 
didactic role on his or her part (i.e. a guide for individualized learning). He or she must leam this new role, he or 
she must know what multimedia can bring to the learners and Aus multimedia technology has to be leered. 
Collis (1996 p 22) contends that the teacher has the responsibility for "die eventual success or lack of success 
of any compUers-in-education initiative". Teachers should then be prepared to teach with technology, which is, 
after linguistics, specific fields and didactic theories, the fourth aspect of their training. 

"Don't overdo it !", anybody would say to me at this point Memwhile a fifth competence is needed 
because multimedia technology makes it indispensable now. Individualized teaching calls for a multimodal 
interlace that would suit the whole set of learning attitudes that cognitive sciences have clearly characterized for 

Cognition is thus another field of competence for any teacher today that cannot be ignored. "Who still 
wants to be a langiiagR teacher ?" Positive answers will probably come fixim the ones who would take pleasure 
in seeing their students taking pleasure by woridng with their own customized software ! 



The concept of "Mediatizing" 

The word "cognition" comes fimn the works of Plato and Aristotle. But cognitive sciences have only 
arisen since the 1950s (the MIT Symposia on Information Theory) and now are of what philosophers call 
the unity of Science: « perhaps as a consequence of the ever-increasing specialisation in science, scientists have 
found it important to woric in an interdisciplinaiy manner where they can draw iqion the research skills and 

knowledge bases of scientists trained in other disciplines » (Bechtel, 1998). 

This necessary interdisciplinarity is more and more observed after the specialisation recommendations 
extolled at the beginning of the 70s. It now allow us to have a heuristic altitude towards any mental pnxiuctioa 
Before try ing to imagine how to implement a didactized content according to different cognitive modes, it is 
essential to consider the status of this content It is in feet something tiie teacher has m mind and something he 

or she has to express. At least a general idea of how things worirfixm mind to expression may be useful. 

Kathryn Bock has characterized this process in an interestingly synthetic way. « Psychohngu^c 
research on language production concerns itself wife fee cognitive processes that convert non verbal 
communicative intentions into verbal actions. These processes must translate perceptions or thoughts mto 
sounds using fee patterns and elements of a code that constitutes fee grammar of a language. For theories of 
language production, fee goal is to explain how fee mind uses this code when converting messages mto 

sptmtaneous speech in ongoing time » (Bock, 1995). , , ,, i- • 

A lot of research is being done in that field now (Pottier, 1992) and a theory is needed to formalize m a 
hierarchized and dynamic way fee construction of representations drawn fiom experience and fee pCTSon^ 
calculations completed on feese representations. Artificial intelligence or language translators wofed need su^ 
a theory and cannot go on using words instead of concepts. A concept c^ot be represer^ by a word, 
necessarily a discrete element in a particular language. Concepts belong to mind and have no lin^stic TOloin. 
They may be represented by codes (Toma, 1984, p. 279) and feese codes should then be organised accordmg to 

different categories. . ^ 

It is not possible here to develop this dieory, still under constructioa The theory has nevw^less 

nothing to do widi chronogenesis and does not try to reproduce what happens in the human mind. M^cal 
researchers may one day find it out, but as fer as cognitive scientists are concerned, fee construction of what 



' PC Expert (November 1999, p. 168-174) has tested four leading translator systems: Reverse (ww-softis^o.com) , 
Systran (www.svstransoft.coml . Transcend 2.0 ( www-ttanspaient.can) and Power 'I^'ator (www.tel,^\ N ome of 
them could propwly transit in French the sentence " AM God said, 'Let there be hght, and there was hght . The correct 

translation is: "Et Dieu dit, 'Que la hnnite soft,' et la hnnifere fOt". . j:. n -o.. 

The obtained translations were respectively "Et le Dieu a dft, laisser fe Stre 16ger,' et il y avait l^ger* , Et Dieu fets, 
fe 16eer ' et il y avait lumifere", "Et Dieu ont dit, laisser il y a 16ger,' et il y avait 16ger" and 'Et Dieu ont <ht, il y ait himfere, et 
il y avtit 16ger" ! None of these translations make sense. The imperative form has never been transited ^p^ly and a 
general mistake has been made on the inteipretation of light as an a<tiective ("16ger") instead of a noun ("lumitae ). 
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hap^ in this "black box" has just to be imagined and tested through experiments. A brief outline of this 
construct cm reveal a possible structure starting fiom a small number of metaconcepts, or "meaning atoms", 
^t assemble m ^cepts (or molecules) according to the type and strength of the relations between them, then 
m groins of mol^ules (basic ideas) and in conceptual networks (sets of ideas) according to the different 
calculations occurring on these elements under specific conditions or goals^. 

According to Sperber, « mental representations are themselves objects in the world, and therefore 
potei^al objects of second-ordCT representations or metarepresentations » (Sperber, 1999). So the task involved 
IS in fact an artificial objectivation of subjectivity. 

if metacognition is not a new science (James, 1 890), the important point here is to clearly identify 
the froitoer between the Tilack box' (what is in mind, what is processed in mind, what is on the point of being 

exjTOssed under a certain form or medium, still in mind), and what is expressed under a particularly chosen 
medium ^ 

TTiis medium can be sound for oral language and music, image for stiU pictures, films, painting and 
evOT wn^, movement for dance, mimics and Ixxfy expression, or several media linked together for a 
multimedia siqiport (talkmg movies, television or multimedia computer). 

P^sage, this act of expression is what I caU "mediatizing' ("Mise en m6dia" in French) 
M^atizmg for me is a semiotic inscription. This tenn will NOT refer to the classical way of translating some 
mnerent media (images^ texts, sounds) already edited in a discrete way into a multimedia fonn ('TS46diatisation*' 
m Fnmch). On the contrary, it consists in the direct use of any medium, that is of any way of expressing 



The concept of "Multimediatizing" 



When a multimedia support is chosen to ejqjress a didactized informational content, the act of 
expression can therefore be called "multimediatizing'. Multimediatizing is just a specific form of mediatizing 
evOT It It encompasses three different media; oral wording when a language is used under the form of sounds, 

wn^ wordmg when a language is used under the form of letters and still or moving images to serve as a nron 
to the discourse situation. ^ ^ 



to television, cornputerized multimedia offers hypernavigation (direct access to any part of 

tte mfoTMtional content), intera^vity (reaction to any action to assess the learner's tasks or to help him or her 
m case of rarors) and multimodality that I would define as the possibility for the receiver to adapt the reception 
ot each medium to his own cognitive mode (Toma, 1998)^. 

All these possibilities have to be exploited for an enlarged, deeper and richer learning process But 
whOT you think multimedia" to express yourself under a multimedia form, the problem is "Where to start ?". 
As fer as langji^e is concerned, and particularly when English has to be taught to French students, the 

answer is undoubtedly fiom the SOUND and not fix>m the text 



Frrach students learning English usually have a fair level in understanding written Fn g1i«h but would 
be at a ^-beginner level to understand spoken English. The reason is sinqile. Students have memorized the 
wnttOT form of words because it resembles the French written form. The pronunciation of the words, i.e. their 
o^ form, which needs an effort to be remembered because the phonological system of English is conroletely 
different ^ the French, is either ignored or wrongly learned. Phonetics, the only way to write sounds on 
paper such as a score allow it for music, should be taught in high schools but is not, since most students do not 
even imow what phonetics is about after 10 years of English! The University is there to make up for this lack of 
initial knowledge hoping that late is better than never. 



Aur^ comprehension is therefore an urgent heed for most students since they have a of two 

years of Enghsh left (an equivalent of 20 to 40 hours!) before quitting the University. 

Easy access to the sound should then be facilitated. 



T^stonch of rne^gnmon that I might call the « cognUivo-disciirsive approach » should even help us solve linguistic 
V s^ as why m the two semantically close sentences « He went out in spite of the cold » and « the col^d not 

Keep turn trom going out », the constraint laid on the concept of « going out » by the concept of « cold » is expressed in one 
SCT^K! by the prepositional phrase « in spite of » and in the other one by the verb « keep fiom ». 

^ instance to change the size of a picture (larger if he or she is of the visual type, smaller for 

aud^e types), or move it fiom the right to the left if this suits him/her better, but these possibilities must have been 
anncipateo by the designer. 
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At the beginning of the 90s when sound cards began to be installed in most computers, I imagined a 
new system enabling an automatic segmenting of the sound track according to the blanks or the volume drops m 
the sound signal detected by the system. The background noise can be parametered allowmg a distmcbon 
between what is language sound and what is not, which makes segmenting possible even m case of a 
background music. The value of the blanks to be detected can also be parametered (fiom 0.2 second to 8 
seconds). A "hard" segmenting rate of the sound track at a value of 0.2 second will give a large number of shcrt 
sound segments (one word or more) to suit weak levels of students whereas a medium rate of 0.5 second will 
give a smaller number of longer sound segments for average-level students. Values above 1 second are simply 

used to segment a sound track in large paragraphs. 

I called the system "UVAC®" for "Laboratoire Audio-Visuel Acbf-Comparatif (Audio-Visual 
Active-Comparative Laboratory). This system which was first an authoring system gave birth to a student 
interface and a teacher software for distance and real-time tutoring in a computerized laboratory. The LAVAC 
system Has been used in more than 150 umversities in France and abroad since 1992. 

Moreover during the automatic segmentation process, the system also creates an answering time span 
after each segment proportional to the length of each created segment The proportional value can also 
parametered in a 10 to 999 % range. I usually use a 150 % value, which means for instance ftiat there is a 3- 
second answering time after a created sound segment of 2 seconds. This answering time can eithw be used as 
an automatic pause in aural comprehension for a better understanding process (because slower) or to note 
taking But it can also be used for repetition of each segment, which is nrt as easy as one might fliink, smce 

comprehension, memorisation and language production are involved in liini^ time. 

Nevertheless few courseware programs take into account the notion of a time-liimt for an answCT. 

« Only recently has temporal perception become a central issue again, because cognitive processes cannot te 
understood without their temporal dynamics » (Poppel, 1997). But this notion is fimdamental to prepare the 
learner to react quickly in a virtual environment in order to make him/her ready for action in a real one 

The sound track can of course be listened to without answering time and iq> to 24 ways of accessing 
the sound are possible in a LAVAC lesson with just a simple real-time segmenting of the sound track Tte 
visualisation of the sound spectrum can be proposed for a better peiceptiOT of the sound But most importantly 
the sound segments are automatically numbered and visuahsed Even if a segment selection is played m a 
listening or recording mode, any of them can instantly be accessed by a right click of the mouse. Any type of 

segment selection (continuous or discontinuous) is also possible. 

These 24 ways of accessing the sound will obviously suit the different cogmtive types of lexers. Not 
surprisingly, the analytic type will listen to the sound track segment by segment, the synthetic will preto a 
global listening of segments and the kinesic will listen to groiqis of segments. It is importmt to note tot to^ 
different ways of accessing the track are generally independent fiom the learner’s level or the difficulties of the 

material to be listened to. i. • 

In the LAVAC system, each numbered sound segment is in fact a sequence that can be given a name 
and tot can be linke d to a text (and/or a text answer), one or several images, a video and all tyi^ of ei^^. 
In case of an error, an hypermedia link to one or several sequences of another lesson for revising or 

reinforcement can be set iq> by a mouse click. i * 

Since all there possibilities can easily be inqilemented by a non-programmmg teacher, it is essential at 
that point to have minimum of knowledge of what cognitive science and particularly activity th^es can brmg 
us in the understanding of a learning process: the subject becomes the product of his or her own interactions m a 

conscious learning process. , , 

Leontiev in 1972 posed three hierarchized levels of relations between the learning subject and the g<^ 
to be attained. The first is the intentional phase that defines the need to be satisfied (imiTOved knowl^e for 
exan^le), the second, the strate^es to be implemented and the third, the specific routines or acts tot will 

involve and put into use pre-existing knowledge procedures. ' c a. 

Bruner in 1990 adds a recursive cycle at each step of this structure for assessment of the rraults 
obtained. The analysis of the discrepancy between the expected and the observed effects is the « fuel » of the 
dynamics of the system (feedback and auto-correction processes) until final satisfection ot fsabm. 

The key phrase in language learning in a virtual environment is probably 'lea^s contol to an 
accepted effort" The learner will better mobilize and stimulate his or her mental and cognitive abilities « “e or 
she has a personal chaUenge to win. But he or she must be sure to win it with the necessary tools provided by 

the teacher. ... * ^ • i ^ 

Thus allowing an easily accessible solution with no time-lmut to answer is certainly not me nght 

solution to improve learning through multimedia teaching. 
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Videos that can be played with a "tickable" transcript option should then be avoided since aural 
c^I^hension is then transformed into understanding of oralised written English. At least this possibility of 
help for weak learners should come after a certain number of errors in the exercise sectioiL 

In the same way some (rare?) method designers should have a better epistemological insight of what 

asked after a video played without any help of any sort cannot be 
answered if the sound ^ not been understood But when multiplwihoice questions are proposed the learner 
can have the averse if he is lucky! Is this still language learning ? What is the point of knoSng tiSt an answer 

“»‘^erstan‘l why? If on the other hand the sound has been properly understood 
questons shoidd be better rased m a non-virtual environment with the teacher and the student alone or in stnall 
groups to check memorisation and favour oral expression and reuse of learned vocabulary. 

In fart a machine must be used for what it can do best: tireless playing of the sound track with always 
fte s^e quahty segmOTtation of this track, recordings, listening of recordings, visualization of segmente 
visualmtira of the sound tack, comparisons of spectrograms, insertion of indexes in the segments for shorter 
ones It needed, recap functions for immediate listening fixMn any index, role play, etc 

" why I desired two new tools derived fiom LAVAC to help non computer-literate teachers 
create then own specific software m real time: the Audio Sequencer (Virtual record^®) in 1998, and the Video 
... , tools may also be used by teachers who consider that computer exercises are lone 

to impteOTt and can ^vot to the student in a "normal" session in a traditional classroom. In some cas^ 

ac^ss to the lang^e 1^ is limited because of the large number of groins who want to use it, and again it is 
wise to choose to do with a machine what is impossible to do in the classroom. ^ 

But these tools could nevertheless be linked to the LAVAC student interface to give them what the 
m^hme done ^ot process: different levels of help. Help, missing in most software applications, are what 
^ student to find the solution by him or herself when they are presented in a graded way There is 

no need of a Bayesian approach to distinguish between imposed hints, necessary for all the students and 
proposed o^ only for those who need ftem or to assess the number and types of hints to be implemented. 

valiMS ofthe automatic segmenting enabled the same document to be used for different levels 
of ^ different levels of help in succession will make any document appropriate to a wide 

range of studente, as long as there are no beginners or semi-beginners. 

aim in this approach is to base the software application more on the implementation of 
metalinguistic activities than on a classical acquisition of lexical or syntactic data. 

^ tojje ^ved (with the problem of time) is then to implement the metalinguistic lea rning 

different cognitive types, knowing that each learner more or less belongs to 
of metalinguistic activities shall be introduced and most of these are based on the 
nypotoebco-deductiye techmques the learner commonly uses in everyday-life deductive reasoning. 

for sound discrimination and word recognition: a hierarchized list of different deduction help starting fiom 
wave spectrograms, phonetic transcription, lexical hints, written form of words 
- for word and sentence meanmg: etymology, knowledge of the discourse situation (solve a linguistic 
problem ^mg an extralmpi^c element of knowledge), contextual logic (the sentence is considered as in a 
crossword gaine where missmg words are missing letters, and the context gives the clues) will be put into 
use to e^le the learner to find the meaning of the words or the gioi^ of words by him or herself 
role of fte image medium in understanding: an image can seldom be used to illustate one particular word. 

n understanding of the current language situation, especially if it is authentic. The 

IMTECT w^ not have to wmider about the place, the look of the characters, even their self-ejqjtession, and 
all sorts of elements that might perturb him or her in his/her understanding process (Toma, 1996). 



The concept of "Re-mahmiediatizing" 



Even if different level segmenting of the sound track has been completed and a whole set of gap-filling 
OT matc^ raerc^ and multipleKjhoice questions with hypermedia links in case of error has been devised, 
the work of the authoring teacher is not over. 



See Toma, T. (2000). Real-time course design: the LAVAC Video Sequencer. SITE 2000, San Diego, Ca 
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Even though or a range of possible help has been implemented with a semiological code coherent 
throughout the courseware, the authoring teacher should better go into the language laboratory to see what 

happens. * 1 . t* * 

Cognitive types (analytic, synthetic, Idnesic) are one thing, personahty types are another. It is 

necessary to be in the language lab to observe different learning attitudes, most of them based more on work 
evaluation than on knowledge acquisitioa The cautious or unadvanturous type will refr^ fixm imaging any 
type of solution. The hints may be modified to make things easy and avoid any logical risk. The confident type 
on the contrary might not see the traps. Question marks, specific color codes should draw Ins or her attei^on to 
make him avoid simple mistakes. Ofliers may be of a passive type. They want the solution immediately. Si^ial 
tutoring is then needed, which is possible when a communication area has been desired in the language lab. 

CD-ROM programs with a fixed content designed for a hypothetical didactic situation cannot be 
satisfactory then with these methodological prerequisites. The necessity to meet specific user requirements ^ts 
stronger and stronger. LAV AC software allows precise follow-iqi of each learner’s activity and all sorts 
and easy modifications are possible even in the presence of students; change of answeimg time for a whole 
group for any lesson in case of different student level, modification of the size or position of one ot severe 
images or texts thanks to multiselection tools (the set-vqi for one will be ^lied to all »lec^ elements), 
addition or siqipression of one or several questions, change in marking, creation of hypermedia hnla, etc. ^ 

This concept which few cognitive scientists are concerned about is what I call "re-multimediatamg . 



Conclusion 

Thanks to the familiar cognitive operations implemented by this type of methodology, the learner will 
be able to successfuUy identify the new linguistic and extra-linguistic data he or she has progressively acquired, 

in relation with his or her pre-existing knowledge network. . u . 1 . ^ 

The asf^gsm«mt of the unconscious mental organisation of the acquired data should ften become 
conscious through final testing in order to improve it if possible, with the help of &e teacher, rf ne^ssaty. 
T an gnagff skill s and knowledge acquisition will then normally occur through 'participative coiqilmg (Varela, 
1 994) and representation construction. 
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